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Teleoperators are the humans who control devices from a “distance.” This distance might be 
extreme physical separation, as in remote assembly of Space Station Freedom by ground-based per- 
sonnel. But the distance could be effective distance of scale, for the manipulation of microscopic 
structures like single living cells or the components within an integrated circuit chip, or even the 
conceptual distance of safety, where the devices effect their actions inside a nuclear reactor s core or 
in the ocean’s depths. Virtual environments refer to the synthesized realities that can be generated by 
various types of computerized displays, not only visual displays but also acoustic, tactile, and force- 
reflective displays. Video games are common but usually limited examples of virtual worlds. An 
implicit theme is telepresence , a term whose definition is imprecise. It involves the use of virtual 
environments to improve the efficiency of teleoperators by giving them a compelling sense of “being 
where the action is.” Telepresence operations seem more natural and facile, and thus more easily 
trained, than other possible models for the human-computer interface. Consider a complex assembly 
task, controlled by wearing a sensor garment that enslaves a robotic arm to mimic the user s own 
arm movements, and visualized on video and felt by tactile and force-reflective feedbacks, versus, its 
control by typing code into a keyboard with feedback via numerical tables. Telepresence seems to 
endow the user with such a robust mental model that he or she becomes absorbed into the synthetic 
re alit y as though absorbed into a vivid dream. 

The primary challenge for reduction to practice is to develop a paradigm for human-computer 
interaction that will enable telepresence to be implemented reliably. The natural movements of the 
human operator, such as the ability to “look around” in the virtual world, provide an important com- 
ponent for this paradigm inasmuch as the consistency of the virtual world can be enhanced by its 
enslavement to sensed head or eye movements of the human user. 

In the present paper, we explore effects on the human of exposure to a visual virtual 
environment which has been enslaved to simulate the human user’s head movements or eye 
movements. Specifically, we have studied the capacity of our experimental subjects to maintain 
stable spatial orientation in the context of moving their entire visible surroundings by using the 
parameters of the subjects’ natural movements. Our index of the subjects’ spatial orientation was the 
extent of involuntary sways of the body while attempting to stand still, as measured by translations 
and rotations of the head. We also observed, informally, their symptoms of motion sickness. 
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METHODS 


Subjects 

A total of 93 university students or staff, 50 males and 43 females ranging in age from 18 to 
43 years old, participated as unpaid volunteers in one of four experiments. Some students received 
course credit. Subjects were prescreened for medical histories (self-report) and any with possible 
vestibular defects were excused. Any subjects requiring refractive correction used their own pre- 
scribed lenses. 

Apparatus and Procedures 

Head Position Sensing. An acoustic timing principle was used to locate in space the headgear 
worn by each subject. This lightweight headgear contained two click sources, the wavefronts from 
which were detected with microsec timing resolution at each of four microphones attached to the 
ceiling of the testing room. Software algorithms for windowing the microphone outputs and per- 
forming redundancy checks, then implementing solutions of the Pythagorean theorem, permitted 
each click source to be located in space with translation precision below 1mm at a sampling rate of 
50 Hz maximum. Resolution of the rotation angle for the line connecting the two click sources was 
below 0.2 deg (25 Hz maximum sampling rate). The headgear configuration did not permit resolu- 
tion of rotations around the line connecting the click sources (pitch), but did permit resolution of the 
other five degrees of freedom for head movements. 

Visual Environment. Each subject stood near the axis of a vertical hemicylindrical screen onto 
which was projected a pattern of vertical stripes. This pattern subtended in excess of 180 deg of 
azimuth and 120 deg in elevation. The instantaneous azimuth positioning of the pattern was under 
software control with a resolution of approximately 0.04 deg at a maximum update rate exceeding 
lKHz. Impedances within the projection system were compensated for via nonlinear control signals 
when motions simulating saccadic eye movements were employed. In those particular experiments, 
the control signals were precomputed to reduce lag time to below 1 msec. When motions enslaved to 
the subject’s head movements were employed, it was impractical to precompute the control signals. 
Thus, there was an additional lag time for computation of about 150 msec and the update rate was 
reduced to 10 Hz. 

Simulations of Saccadic Eye Movements. Two small fixation lights rear-projected onto the sur- 
rounding screen could be alternated to direct horizontal saccades of particular azimuth extent (0.5, 1, 
2, 3, 4, 6, 8, or 16 deg). Motions of the pattern front-projected onto the screen had characteristics of 
average velocity and motion duration similar to those for each corresponding saccade extent.^ In two 
experiments, the saccade-like stimulus motions occurred independent of the subject’s eye move- 
ments. In another experiment, the stimulus motions were triggered by subject eye velocity, derived 
from the electro-oculogram, when the eye had moved 15-20 aicmin from its initial fixation during a 
guided voluntary saccade. 


3 a. T. Bahill, M. R. Clark & L. Stark, Mathematical Biosciences ; 24, 191-204 (1975). 
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Head Movement Feedback Conditions. Each sample of the head’s fore/aft and left/right 
translations plus head yaw angle permitted real time calculation of an equivalent azimuth change for 
the surrounding visual stimulus. This equivalent azimuth, modified by a gain factor that was the 
parameter selected for study, controlled motion of the visual stimulus in this experiment. The gain 
factors used were +2, +1.5, +1, +0.5, 0, -0.5, -1, -1.5, and -2. The gain of +1, for example, could 
serve in principle to keep the stimulus “straight ahead” by its faithfully tracking the head’s 
movements. Larger gains moved the stimulus farther and faster than would accurate head tracking; 
smaller gains moved the stimulus less far and more slowly. Zero gain made the stimulus remain still 
as in the natural environment. Negative gains reversed the left/right correspondence between head 
movements and stimulus motions. 

Data Analysis 

Control Conditions. Although the stability of a standing human has been modelled as an 
inverted pendulum, since the body’s center of gravity is above its support base, the problem with 
such a model is that the human body is not mechanically rigid. Rather, it is a jointed mechanism 
controlled by over 400 muscles and several sensory feedback loops, with correspondingly complex 
dynamics. Furthermore, people grow to various heights, weights and distributions of body mass; 
with various extent of conditioning for their musculature; and with varied skills of sensorimotor 
integration. Individual variations in the body’s dynamics are, therefore, prominent and, as a 
consequence, we rely on the use of experimental designs allowing each individual to serve as his/her 

own control. 

Also, it is impractical to open the feedback loops from proprioceptive/kinesthetic or vestibular 
senses but it is relatively easy to alter those loops by asking the subject to stand only on one foot or 
else on a compliant surface. As reported in the results below, there are multiple techniques for con- 
trolling the visual feedback loop within each subject, including: (a) injecting noise via random 
motions of the visual environment, (b) stabilizing the environment via motion feedback from head 
movements, (c) eliminating contour from the visual environment, or (d) eliminating visibility by 
turning off the lights and (e) by closing the eyes. 

Within-Subject Figure of Merit for Body Stability. There is no consensus in the literature for a 
representative figure of merit to describe such complicated behavioral dynamics as those under con- 
sideration. We have not been satisfied, in general, with time-domain analyses such as average 
movement velocity or RMS position error, since these statistical summaries tend to obscure details 
of the responses which we find to be correlated with experimental manipulations. We have, however, 
found one particular form of time-domain analysis to be useful in certain circumstances, and that 
form is time-locked response averaging. We plan to explore in future work the more general utility 
of a “chaotic” time-domain (phase-sensitive) analysis. 

We have enjoyed considerable success in accounting for individual variations among subjects by 
using frequency-domain analysis, however. Specifically, each subject is measured: (a) under control 
conditions to determine his/her individual power spectral density (PSD) functions for body move- 
ments over time, and (b) under experimental conditions, to determine any changes in the PSD 
functions induced experimentally. The ratio between PSD functions (a) and (b) is the gain, as a 
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function of the frequency, induced by the experimental manipulation. We find these gains to be 
relatively consistent betwen individuals (standard errors of measurement typically less than 1 dB). 


RESULTS 


Figure 1 shows sample records of two subjects’ lateral head movements while viewing moving 
surroundings (middle two traces). For the top trace, the motion of the stimulus was saccade-like, 
while for the bottom trace, the stimulus tracked the subject’s head movements (gain = +1.5). These 
sample records have been displaced vertically for clarity. 

The top trace of Figure 1 shows typical lateral sways made by subject S 1 when the entire visible 
surroundings suddenly moved with the velocity (66 deg/sec) and duration (30 msec) characteristic of 
a 2 deg saccadic eye movement. Note the relatively pronounced fluctuations in head position. By a 
comparison against the second trace, S l’s lateral sways in still surroundings, it is clear that the stim- 
ulus motion exerts a strong destabilizing effect. 

The bottom trace of Figure 1 shows lateral sways made by subject S 2 while the azimuth position 
of the visible stimulus tracked S 2’s own head movements. By comparison against the third trace, 
subject S 2’s lateral sways in still surroundings, the destabilizing effect of head tracking is apparent. 
Individual variations in stability can be noted by comparing the third trace (S 2) against the second 
trace (S 1), sways made in still surroundings. 

Figure 2 depicts averaged yaw movements (i.e., head rotations) made in reponse to saccade-like 
motions of the stimulus, plotted as a function of time after the motion began. Each plotted point is 
based on 10 repetitions per subject times 18 subjects, or the average of 180 measures of head yaw 
angle per condition. The measures were carried out in three conditions: (1) saccade-only, the subject 



Figure 1. Sample records of head movements in still vs. moving environments (saccade-like or head 
tracking motions), for two subjects. 
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Figure 2. Response averages showing head yaw elicited by saccade-like stimulus motions. Motions 
triggered by saccades are less effective than untriggered motions for eliciting these responses. 

moved their eyes as directed by fixation lights but there was no additional motion of the stimulus, 

(2) motion triggered by the saccade, in which a saccade-like stimulus motion took place while the 
subject’s eyes were moving in a saccade, and (3) untriggered motion, in which a saccade-like 
s timul us motion briefly preceded (<100 msec) the subject’s execution of a saccadic eye movement. 

In the saccade-only condition, subjects showed a slight tendency to yaw the head as well as to move 
the eyes. Average head yaw for saccade-only is taken as the baseline in this plot. This baseline 
motion is subtracted from the head yaws in response to stimulus motions. 

It is clear in Figure 2 that untriggered saccade-like stimulus motions elicit a small but reliable 
yaw (the peak at 0.5 sec represents 10 standard errors difference). This may be qualitatively similar 
to the lateral head translations illustrated in the top trace of Figure 1. However, Figure 2 also shows 
that the effect of the saccade-like stimulus motions can be lessened dramatically by causing them to 
coincide with the subject’s own voluntary saccades. In this latter case, possibly a signal generated 
internally by the subject’s oculomotor system (viz., corollary discharge or efference copy) alters the 
consequences that otherwise accompany the stimulus motion. 

Figure 3 is a plot derived from fast Fourier transforms of sway records analogous to those shown 
in Figure 1. For the plot shown here, the power spectral density function (PSD) for each subject was 
obtained in still surroundings and this was taken as the baseline for that individual. PSDs for that 
subject under other conditions of stimulation were divided by his/her own baseline PSD. This 
resulted in gain profiles (dB relative to baseline) as a function of sway frequency (Hz). Each point on 
Figure 3 represents the average gain across 24 subjects (18 for saccades condition) within a fre- 
quency band 0.5 Hz wide. Typical standard errors associated with each point are under 1 dB. 

The six curves shown in Figure 3 depict the effects on subjects’ stability of (1) closing their eyes, 
(2) e liminatin g visible contour (subjects wore diffusing goggles), (3) eliminating illumination 
(subjects wore opaque goggles, (4) moving the azimuth of the surroundings to track the subject s 
head movements (gain ±1.0), (5) moving the azimuth of the surroundings randomly, and (6) the 
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Figure 3. Comparisons of the gain in lateral sway as a function of sway frequency, for experimental 
and control condition. 

subjects made directed saccadic eye movements across the still surroundings. Two of the curves on 
Figure 3 were moved vertically to reduce overlap: the “lights out” curve was moved down 2 dB and 
the “head track” curve was moved down 3 dB. “Eyes closed” caused significantly larger sways than 
either “no contours” or “lights out,” which did not differ reliably. “Head track resulted in signifi- 
cantly smaller sways than in those conditions. 

All six curves show a general trend of decreasing gain with increasing frequency, particularly so 
below 2 Hz. To a first approximation, the gain profile associated with any of these six conditions 
appears nearly interchangeable with the profile from another condition by simply shifting the curve 
up or down on the graph. Since all of these conditions might serve to reduce or degrade visual feed- 
back from the environment, this first approximation similarity is not overly surprising. In detail, 
however, it is interesting to note that the saccades condition (6) actually made the subjects sway less 
than did fixating in still surroundings. Also note that the head track condition (4) is the only curve 
with a large gain decrease at 0.75 Hz (midpoint for the 0.5 to 1.0 Hz band). The frequency depen- 
dence of gain profiles with head tracking is more apparent in the following graphs. 

Figures 4 and 5 compare the effects of four different gain factors for head tracking. The gain 
factor parameter relates azimuth positioning of the visible surround to the combined fore-aft and left- 
right translations plus yaw angle sensed for the head. Each point on these figures represents the aver- 
age movement gain across 33 subjects, calculated within 0.5 Hz bandwidths. These gains, the ratios 
of PSD profiles in tracking vs. baseline conditions, were referenced to “random motion” baseline 
measures with corresponding gain factors. These baselines were necessary because the visible sur- 
round exhibited a slight random jitter due to system noise even when attempting to track a fixed arti- 
ficial head. 

There are two noteworthy features shown in Figures 4 and 5. First, the gain profiles are clearly 
not flat but instead rise or fall in a frequency-dependent manner. Thus, we must conclude that mov- 
ing the visual stimulus by tracking the subject’s head differs qualitatively from the various other 
forms of altered visual feedback depicted in Figure 3, because those manipulations have yielded 
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Figure 4. Lateral sway gain as a function of frequency when the azimuth of the surrounds tracked the 
subject’s head movements. The four curves represent different gain factor. 



Figure 5. Yaw movement gain as a function of frequency, analogous to Figure 4. 

results that are highly similar across frequency. Second, whether each curve in Figures 4 and 5 nses 
or falls in each frequency band is highly dependent on the sign of the gain factor parameter. For both 
lateral sway (Fig. 4) and yaw movements (Fig. 5), a gain peak associated with positive parameter 
values becomes a gain trough when associated with negative parameter values, and vice versa. 

Even though subjects in the head tracking experiment showed greater stability than did subjects 
who simply closed their eyes, 10 out of the 33 reported symptoms of motion sickness. For 4 of those 

subjects, symptoms persisted more than an hour, and for one subject, persisted 5 hours. Another 
8 subjects found the participation unpleasant but not nauseogenic. In contrast, 3 subjects reported 
extreme exhilaration apparently akin to taking an amusement park ride. Only one subject guessed 
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that the stimulus tracked her head movements, while another subject guessed that her head had been 
captured by motion of the visual stimulus. 


CONCLUSIONS 


We find that exposure to a visual virtual environment enslaved to simulate a subject’s own eye 
movements or head movements can have deleterious effects on the subject’s spatial stability. When 
the entire visible surroundings were moved with the same parameters of angular velocity and motion 
duration which characterize natural saccadic eye movements, subjects tended to lose lateral stability 
(Fig. 1) and to make head yaw movements (Fig. 2, untriggered motions). This pattern of response 
was cancelled, however, when saccade-like stimulus motions were made to coincide with the 
subject’s own voluntary eye movements (also in Fig. 2, triggered by saccade). It seems plausible to 
interpret these findings in the light if theoretical frameworks which propose that a subject generates a 
hypothetical internal signal regarding their own voluntary eye movements. Such an internal signal 
(corollary discharge or efference copy) would presumably serve in the natural world to mitigate the 
effects of retinal image motion accompanying one’s voluntary eye movement. In the present virtual 
world, the same hypothetical internal signal presumably mitigates the effects of external stimulus 
motion if it is made at the same time when this internal signal is active. 

The effects on a subject’s spatial stability when the visual surroundings tracked head movements 
are less readily explained. First, any subject intentions to move the head were discouraged, inasmuch 
as (a) they were instructed to stand still and (b) they became significantly at risk of falling if they 
moved the head voluntarily. It must be questioned whether an efference copy or corollary discharge 
internal signal could exist without subjects’ internal plans to make voluntary movements. Second, 
subjects’ movement gain profiles in Figures 4 and 5 are reminiscent of the changes in poles and 
zeros in the complex domain when a device receiving feedback undergoes a change in its feedback’s 
sign. It is tempting to speculate that head tracking conditions effectively reveal some of the subjects’ 
own internal feedback mechanisms. Unfortunately, the lag time required to update stimulus azimuth 
after sensing head position and orientation (150 msec) was sufficient to complicate detailed interpre- 
tations along these lines. Third, it may be important to understand why so many subjects in the head 
tracking experiments experienced motion sickness symptoms or other discomforts, even though they 
did not become very greatly disoriented. We conjecture that the head-tracking visual feedback must 
have been in conflict with the subjects’ own internal models of their orientation in space. Such 
models are probably multisensory, though highly dependent on visual feedback, and probably estab- 
lished below the level of one’s conscious awareness. 

We believe that the design of virtual environments which are enslaved to the eye or head move- 
ments of a teleoperator must take into account the effects on the human. In the present experiments, 
we found that these effects can include loss of stability, motion sickness, or other undesirable influ- 
ences on situational awareness. These presumably could be mitigated by carefully matching sensor 
resolutions and real time controls of the virtual environment to the capacities of the human 
participants. 
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